import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import folium
import us

1. Introduction

In the middle of January 2020, the COVID-19 pandemic struck the world and the United States and has been terrorizing the U.S. ever since. Everything from shops to schools were shut down to help slow the spread of this new virus. Normality has been shifted from being social to staying home all day and all night. At the beginning of the pandemic, in 2020, the uniqueness of a federal government holding loose authority over a multitude of states, was laid bare. The culture and politics of each state has played an important role in shaping the decisions states made during the waves of infection and now that vaccines are widely available, seem to affect vaccine hesitancy. Vaccine hesitancy describes the behavior when individuals choose not to get vaccinated, despite the vaccine being available, free, and accessible. Epidemiologists are worried that hesitation to get vaccinated may delay or make it impossible to beat back the pandemic.

The goal of epidemiologists is to achieve 'herd immunity', where enough people are immune and will slow the spread of transmission, should it occur. Given the transmisability of COVID-19, scientists indicate that to achieve herd immunity, 70-85% of the American population must be vaccinated, or immune to the COVID-19 virus. This 70-85% milestone is no easy feat. As of the date writing this (5/17/2021), 38% of the U.S. population is fully vaccinated (123 million people) and 48% of the population has received at least one dose of vaccine (some of whom are hence fully vaccinated), according to Our World in Data (https://ourworldindata.org/covid-vaccinations?country=OWID_WRL). The issue is, will enough Americans continue to seek vaccine to prevent further widespread disease and dangerous mutations leading to new strains of resistent virus that could take us backward? Scientists and government officials are seeking to understand "why" people are not enthusiastically queuing up for shots.

In this notebook, we will explore the reasons for vaccine hesitancy by state. The data analyzed is publicly available from the U.S. Census Bureau Household Pulse Survey Week 28 (https://www.census.gov/data/tables/2021/demo/hhp/hhp28.html#tables).

0) Initial preparation included: getting data from the CSV file to create the data fram, replacing empty values with zeros, and changing the State field to be the first column in the data frame.

The data was formatted in a not-so-computer friendly way. I needed to reformat and transform the data in order import it and make it usable. The Excel file that I downloaded also was structured with different worksheets per state, so I had to condense the data into a single tab, then I converted the file into a CSV to allow me to import it here.

hes_df = pd.read_csv('Reasons for not recieving.csv')
diag_df = pd.read_csv('Covid diagnosis.csv')

hes_df = hes_df.replace('-', '0')
diag_df = diag_df.replace('-', '0')

cols = list(hes_df.columns)
a, b = cols.index('Reason'), cols.index('State')
cols[b], cols[a] = cols[a], cols[b]
hes_df = hes_df[cols]

cols = list(diag_df.columns)
a, b = cols.index('Diagnosis'), cols.index('State')
cols[b], cols[a] = cols[a], cols[b]
diag_df = diag_df[cols]

1) Identify the most common reasons for hesetancy in each state.

stateReason_dict = {}

accumulative_lst = []
temp_dict = {}
check_state = hes_df['State'][0]
for index,row in hes_df.iterrows():
    cur_state = row['State']
    
    if row['Reason'] == 'Total':
        continue;
    
    if cur_state == check_state:
        temp_dict[int(row['Total'])] = index%13
    if not(cur_state == check_state) or index == 662:
        keys = temp_dict.keys()
        lst = [temp_dict[max(keys)]]
        for i in range(2):
            del temp_dict[max(keys)]
            keys = temp_dict.keys()
            lst.append(temp_dict[max(keys)])
        accumulative_lst.extend(lst)
        stateReason_dict[check_state] = lst
        
        check_state = cur_state
        temp_dict = {}
        temp_dict[int(row['Total'])] = index%13

2) Graph the frequencies of the top 3 reasons across all states

s = pd.Series(accumulative_lst)
s.value_counts().plot(kind='bar',ylabel='Frequency',xlabel='Reason ID',title='Frequency of Reasons Within the Top 3 Reasons for Each State')
plt.show()

1 Analysis: Top Reasons for Vaccine Hesitancy across All States

The top 3 reasons for vaccine hesitance were extracted from each state, then compiled to see their frequency of occurrance.

Reason IDs for the graph above in order of frequency:

1: Concerned about possible side effects
6: Plan to wait and see if it is safe
9: Don’t trust COVID-19 vaccines
10: Don’t trust the government
11: Other
7: Other people need it more right now
3: Don’t believe I need a vaccine
2: Don’t know if a vaccine will work
4: Don’t like vaccines
5: Doctor has not recommended it
8: Concerned about the cost
12: Did not report

The top four reasons cited across the US (1, 6, 9, 10) are related to people's lack of trust in the safety of the product. This is reflective of the polarized and political climate where truth and fact varies by its source and channel. The bottom four options (4, 5, 8, 12) did not appear in the top three choices for any state.

2. Vaccine Hesitancy Reasons by State

It will be helpful to use a US map to see if there are any regional patterns associated with vaccine hesitancy reasons. In this section, I will build a map with a pin marking each state, upon which the top 3 reasons for vaccine hesitancy may be posted. To further visualize patterns in the data, I'll apply a pin color to indicate the number one reason for vaccine hesitancy in a state.

3) Assign coordinates so a pin can be dropped in each state

def getCoords(state):
    c = []
    if state == 'AL':
        c = [32,-86]
    elif state == 'AK':
        c = [66,-152]
    elif state == 'AZ':
        c = [34,-111]
    elif state == 'AR':
        c = [34,-92]
    elif state == 'CA':
        c = [37,-120]
    elif state == 'CO':
        c = [39,-105]
    elif state == 'CT':
        c = [41.5,-73]
    elif state == 'DE':
        c = [39,-75.5]
    elif state == 'FL':
        c = [29,-82]
    elif state == 'GA':
        c = [32,-83]
    elif state == 'HI':
        c = [19.6,-155.5]
    elif state == 'ID':
        c = [43.7,-114.7]
    elif state == 'IL':
        c = [40, -89]
    elif state == 'IN':
        c = [40,-86]
    elif state == 'IA':
        c = [42,-93]
    elif state == 'KS':
        c = [38.5,-98]
    elif state == 'KY':
        c = [37.5,-84.4]
    elif state == 'LA':
        c = [31,-92]
    elif state == 'ME':
        c = [45,-69]
    elif state == 'MD':
        c = [39.4,-76.5]
    elif state == 'MA':
        c = [42.4,-71.6]
    elif state == 'MI':
        c = [44,-84.6]
    elif state == 'MN':
        c = [46.6,-94.7]
    elif state == 'MS':
        c = [33,-89.6]
    elif state == 'MO':
        c = [38.6,-92]
    elif state == 'MT':
        c = [46.9,-110]
    elif state == 'NE':
        c = [42,-100]
    elif state == 'NV':
        c = [40,-116]
    elif state == 'NH':
        c = [43.6,-71.4]
    elif state == 'NJ':
        c = [40.2,-74.7]
    elif state == 'NM':
        c = [34.34,-106.12]
    elif state == 'NY':
        c = [43.06,-75.58]
    elif state == 'NC':
        c = [35.5,-78.5]
    elif state == 'ND':
        c = [47.5,-101]
    elif state == 'OH':
        c = [40,-83]
    elif state == 'OK':
        c = [35.5,-97.5]
    elif state == 'OR':
        c = [44,-120]
    elif state == 'PA':
        c = [41,-77.5]
    elif state == 'RI':
        c = [41.63,-71.6]
    elif state == 'SC':
        c = [34,-81]
    elif state == 'SD':
        c = [45,-101]
    elif state == 'TN':
        c = [35.8,-86]
    elif state == 'TX':
        c = [32,-99]
    elif state == 'UT':
        c = [39.5,-111.5]
    elif state == 'VT':
        c = [44.15,-72.74]
    elif state == 'VA':
        c = [37.7,-78.2]
    elif state == 'WA':
        c = [47.5,-120.7]
    elif state == 'WV':
        c = [38.75,-80.7]
    elif state == 'WI':
        c = [45,-89.58]
    elif state == 'WY':
        c = [43,-107]
    elif state == 'DC':
        c = [38.913,-77.016]
        
    return c

4) Define colors for each pin representing distinct vaccine hesitancy reasons

def getColor(reason):
    c = 'black'
    if reason == 1:
        c = 'blue'
    if reason == 2:
        c = 'red'
    if reason == 3:
        c = 'yellow'
    if reason == 6:
        c = 'green'
    if reason == 7:
        c = 'purple'
    if reason == 9:
        c = 'orange'
    if reason == 10:
        c = 'pink'
    if reason == 11:
        c = 'brown'
        
    return c

5) Create a visualization canvas of the US map

m = folium.Map(location=[38.0000, -97],zoom_start=4,min_zoom=4)
folium.GeoJson('gz_2010_us_040_00_500k.json').add_to(m)

<folium.features.GeoJson at 0x207d1ec6fd0>

6) Draw the US map with pins at the specified coordinate locations and color matching the top reason for vaccine hesitancy

for k in stateReason_dict.keys():
    coords = getCoords(k)
    folium.Marker(location=coords,
                  popup=stateReason_dict[k],
                  tooltip=us.states.lookup(k).name,
                  icon=folium.Icon(color=getColor(stateReason_dict[k][0]))).add_to(m)
m

Make this Notebook Trusted to load map: File -> Trust Notebook

2 Analysis: Reasons for vaccine hesitancy by US state

There are two reasons that are most often the number one reason for vaccine hesitancy across each state. First, people are (1 - blue pin) Concerned about possible side effects and (6 - green pin) they Plan to wait and see if [the vaccine] is safe. There is a high level of consistency of the response. Given that COVID-19 has been shown to be deadly, it is surprising to me that the most common, number one reason for not getting the vaccine is because of fear of side effects. The common side effects are generally mild and of short duration < 1 day.

Given this homogeneity, perhaps by digging into the data we can find differentiators or other variables that may expose differences in the reasons provided for hesitancy.

3. Hesitancy Reasons among those Previously Infected with COVID-19

I hypothesized that the reason one might provide for hesitating to get vaccinated could be related to whether an individual was previously infected with COVID-19. To explore this hypothesis, in this section, I filtered the data to include only individuals who tested positive for COVID-19 at some point in the past.

7) Filter results to include only individuals previously infected, and then tally the top three reasons for hesitance from among this subset.

stateReason_infected_dict = {}

accumulative_lst1 = []
temp_dict1 = {}
check_state = hes_df['State'][0]
for index,row in hes_df.iterrows():
    cur_state = row['State']
    
    if row['Reason'] == 'Total':
        continue;
    
    if cur_state == check_state:
        temp_dict1[int(row['Previously infected'])] = index%13
                                                                                    
    if not(cur_state == check_state) or index == 662:
        keys1 = temp_dict1.keys()
        lst1 = [temp_dict1[max(keys1)]]
        for i in range(2):
            del temp_dict1[max(keys1)]
            keys1 = temp_dict1.keys()
            lst1.append(temp_dict1[max(keys1)])
        accumulative_lst1.extend(lst1)
        
        stateReason_infected_dict[check_state] = lst1
        
        check_state = cur_state
        temp_dict1 = {int(row['Previously infected']): index%13}

8) Graph the top three reasons for hesitancy, from each state, including only responses from previously infected individuals.

s = pd.Series(accumulative_lst1)
s.value_counts().plot(kind='bar',ylabel='Frequency',xlabel='Reason ID',title='Frequency of Reasons Within the Top 3 Reasons for Each State Amongst People Who Have Been Previously Infected')
plt.show()

3 Analysis: Frequency of Hesitancy Reasons among Previously Infected

The three most commonly occuring reasons for hesitancy among previously infected people (1, 6, 9) remain the same as those cited by all people surveyed, however "Other" reason 11 replaces "Don't trust the government" reason 10 in the fourth spot. Again four reasons were never cited in the top three reason lists by state. Unfortunately, this graph does not provide more insight than we had previously.

Reason IDs for the graph above in order of frequency:

1: Concerned about possible side effects
6: Plan to wait and see if it is safe
9: Don’t trust COVID-19 vaccines
11: Other
10: Don’t trust the government
7: Other people need it more right now
3: Don’t believe I need a vaccine
2: Don’t know if a vaccine will work
4: Don’t like vaccines
5: Doctor has not recommended it
8: Concerned about the cost
12: Did not report

9) Create a visualization canvas with the US map

m1 = folium.Map(location=[38.0000, -97],zoom_start=4,min_zoom=4)

10) Find the % of people diagnosed with COVID per state

total = 0
stateDiagP_df = pd.DataFrame(columns =['state','percent'])
for index,row in diag_df.iterrows():
    state = row['State']
    
    if row['Diagnosis'] == 'Total':
        total = row['Total']
        continue
        
    if row['Diagnosis'] == 'Yes':
        new_row = {'state': state, 'percent':float(row['Total'])/float(total)}
        stateDiagP_df = stateDiagP_df.append(new_row, ignore_index=True)

11) Now, use this information to illustrate the saturation of COVID-19 diagnoses on a chloropleth map

folium.Choropleth(
    geo_data = state_geo,
    name ='choropleth',                  
    data = stateDiagP_df,                     
    columns = ['state', 'percent'], 
    key_on ='feature.id',
    fill_color ='Reds',
    fill_opacity = 0.7,
    line_opacity = 0.2,
   legend_name = "Percent of Population Previously Infected with COVID-19"
).add_to(m1)
m1

Make this Notebook Trusted to load map: File -> Trust Notebook

12) Overlay pins, colored according to the number one hesitancy reason in the state, on the chloropleth map for infected individuals.

for k in stateReason_infected_dict.keys():
    coords = getCoords(k)
    folium.Marker(location=coords,
                  popup=stateReason_infected_dict[k],
                  tooltip=us.states.lookup(k).name,
                  icon=folium.Icon(color=getColor(stateReason_infected_dict[k][0]))).add_to(m1)
m1

<ipython-input-484-720ce44134e7>:6: UserWarning:

color argument of Icon should be one of: {'purple', 'darkred', 'darkgreen', 'black', 'lightgray', 'darkblue', 'green', 'lightblue', 'lightgreen', 'darkpurple', 'beige', 'red', 'gray', 'cadetblue', 'pink', 'orange', 'blue', 'white', 'lightred'}.

Make this Notebook Trusted to load map: File -> Trust Notebook

3 Analysis: Hesitancy Reasons among Previously Infected

Now, we start to visually see variation in the hesitancy reasons (according to pin colors by state) for the previously infected. Beyond the common reasons of concern about side effects and a desire to wait and see about vaccine safety, three previously less important reasons emerged across several states. One state, Montana, has infected individuals most commonly citing a distrust of government for the reason why they won't get the vaccine. Two states, Louisiana and Kentucky, has residents saying the don't trust the vaccine. Three states, Florida, Colorado, and Deleware, find infected individuals unsure if the vaccine will work.

There still does not seem to be a correlation between the proportion of infection and the top reason for hesitancy, as evidenced by no clear pattern of top hesitancy reason corresponding to state infection prevalence.

Key: Pin color indicates the number one reason for vaccine hesitancy in a given state.

1: (blue pin) Concerned about possible side effects
2: (red pin) Don’t know if a vaccine will work
3: Don’t believe I need a vaccine
4: Don’t like vaccines
5: Doctor has not recommended it
6: (green pin) Plan to wait and see if it is safe
7: Other people need it more right now
8: Concerned about the cost
9: (orange pin) Don’t trust COVID-19 vaccines
10: (pink pin) Don’t trust the government
11: Other
12: Did not report

4 Hesitancy Reasons among those Previously NOT Infected with COVID-19

In this section, I filtered the data to include only individuals who had NOT tested positive for COVID-19 at some point in the past. This is the converse of Section 3.

13) Filter individuals who did not indicate a previous infection, and includes those who were unsure and didn't respond to the question. Note the logic incorporates the total sample when combined with those who did indicate a previous infection.

stateReason_healthy_dict = {}

accumulative_lst2 = []
temp_dict2 = {}
check_state = hes_df['State'][0]
for index,row in hes_df.iterrows():
    cur_state = row['State']
    
    if row['Reason'] == 'Total':
        continue;
    
    if cur_state == check_state:                                                    # Anyone who did not identify as being
        temp_dict2[int(row['Total']) - int(row['Previously infected'])] = index%13  # previously infected is considered to
                                                                                    # never have been infected                                                                                 
    if not(cur_state == check_state) or index == 662:
        keys2 = temp_dict2.keys()
        lst2 = [temp_dict2[max(keys2)]] 
        for i in range(2):
            del temp_dict2[max(keys2)]
            keys2 = temp_dict2.keys()
            lst2.append(temp_dict2[max(keys2)])
        accumulative_lst2.extend(lst2)
        
        stateReason_healthy_dict[check_state] = lst2
        
        check_state = cur_state
        temp_dict2 = {int(row['Not previously infected']): index%13}

14) Graph the top three reasons for hesitancy, from each state, including only responses from individuals who did NOT indicate a previous infection.

s = pd.Series(accumulative_lst2)
s.value_counts().plot(kind='bar',ylabel='Frequency',xlabel='Reason ID',title='Frequency of Reasons Within the Top 3 Reasons for Each State Amongst People Who Have Not Been Previously Infected')
plt.show()

15) Create the map canvas for visualizations

m2 = folium.Map(location=[38.0000, -97],zoom_start=4,min_zoom=4)

16) Determine the % not previously diagnosed with COVID.

total = 0
stateDiagP_df = pd.DataFrame(columns =['state','percent'])
for index,row in diag_df.iterrows():
    state = row['State']
    
    if row['Diagnosis'] == 'Total':
        total = row['Total']
        continue
        
    if row['Diagnosis'] == 'Yes':
        new_row = {'state': state, 'percent': 1 - (float(row['Total'])/float(total))}
        stateDiagP_df = stateDiagP_df.append(new_row, ignore_index=True)

17) Now, use this information to illustrate the saturation of those not previously infected with COVID-19 on a chloropleth map

folium.Choropleth(
    geo_data = state_geo,
    name ='choropleth',                  
    data = stateDiagP_df,                     
    columns = ['state', 'percent'], 
    key_on ='feature.id',
    fill_color ='Greens',
    fill_opacity = 0.7,
    line_opacity = 0.2,
   legend_name = "Percent of Population Not Previously Infected with COVID-19"
).add_to(m2)
m2

Make this Notebook Trusted to load map: File -> Trust Notebook

18) Overlay pins, colored according to the number one hesitancy reason in the state, on the chloropleth map for not previously infected individuals.

for k in stateReason_healthy_dict.keys():
    coords = getCoords(k)
    folium.Marker(location=coords,
                  popup=stateReason_healthy_dict[k],
                  tooltip=us.states.lookup(k).name,
                  icon=folium.Icon(color=getColor(stateReason_healthy_dict[k][0]))).add_to(m2)
m2

Make this Notebook Trusted to load map: File -> Trust Notebook

4 Analysis: Hesitancy Reasons among those NOT Previously Infected with COVID-19

When we focus only on individuals who did not indicate having had a prior COVID-19 infection, we revert to the norm, since this population comprises the majority of respondents. The same two reasons, (1-blue) a concern about possible side effects and (6-green) having a plan to wait and see if the vaccine is safe dominate in this group.

As in the previous chloropleth for infected individuals, there still does not seem to be a correlation between the proportion of non-infection and the top reason for hesitancy.

Color and Numerical Key: Click pins to see top three reasons for vaccine hesitancy in a given state. Pin color indicates the number one reason for vaccine hesitancy in a given state.

1: (blue pin) Concerned about possible side effects
2: (red pin) Don’t know if a vaccine will work
3: Don’t believe I need a vaccine
4: Don’t like vaccines
5: Doctor has not recommended it
6: (green pin) Plan to wait and see if it is safe
7: Other people need it more right now
8: Concerned about the cost
9: (orange) Don’t trust COVID-19 vaccines
10: (pink) Don’t trust the government
11: Other
12: Did not report

Why is understanding vaccine hesitancy important?

One of the biggest reasons that we should be trying to understand vaccine hesitancy is so that we can decrease vaccine hesitancy. By decreasing vaccine hesitancy, we may be able to get more people to get the vaccine when they would not have otherwise. Expecially right now, learning about vaccine hesitancy is extremely important because this virus is like a ticking time bomb. By delaying when we reach herd immunity, we are giving the virus more time to mutate and become even harder to stop and with so many people being hesitant about getting the vacine, it is very likely unatainable to reach this very important milestone. We have already seen the virus mutate with the UK varient which is much more potent and much easier to spread and we should be doing everything we can to prevent another varient from slipping through the cracks. It is possible that the people who are hesitant to get the vaccine catch the virus and allow it to mutate to a point where the old vaccine is no longer effective, causing a second outbreak to occur around the globe.

Furthermore, if anything, people should be trying to protect themselves. The current form of the virus is extremely dangerous, can perminately damage vital organs like the lungs, and has proven to be extremely deadly with about a 2% mortality rate in the United States according to the John's Hopkins Coronovirus Recource center.

Lastly, we should care about vaccine hesitancy because tackling it can protect the vulnerable people in our communities. This includes the immunocompromised, expecially those who have are living in hospitals, the elderly, and even our babies and children. The success of the vaccine is largely dependent on how many people are willing to get it and how quickly we can distribute doses across the country and across the globe.